Using Conflicts Among Multiple Base Classifiers to Measure the Performance of Stacking

We analyze the machine learning bias of stacking and point out the conflict problem. Conflicts are defined as base data with different class labels that produced the same predictions by a set of base classifiers. Based on conflicts, we propose conflict-based accuracy estimate to determine the overall accuracy of a stacked classifier and conflict-based accuracy improvement estimate to determine the overall accuracy improvement over base classifiers. We discuss some popular metrics for comparing and evaluating a set of classifiers: coverage, correlated error, diversity and specialty, and show that these metrics do not accurately estimate the overall accuracy of a stacked classifier system. From experimental results, we demonstrate that conflict-based accuracy estimate is an effective measure to predict overall performance and compare different stacked systems, and conflict-based accuracy improvement estimate is a good measure to project the overall accuracy improvement.